home *** CD-ROM | disk | FTP | other *** search
- Submitted-by: karish@mindcraft.com (Chuck Karish)
-
- In Volume 31, Number 16, amf@amfent.gwinnett.com (Andy Feibus) wrote:
- >dave@88open.org (Dave Cline) writes:
- >> It seems valid to argue against formal test method standards because:
- >>
- >> c) there is no mechanical way to ensure that both standards
- >> say the same thing.
- >Taking this a step further, a test methods standard may not provide
- >complete or completely correct test coverage of a standard. Following
- >from that, a test suite based
- >on the test methods standard may not properly test the actual technology
- >it intends to verify.
-
- At least if you there is a test methods standard, it's
- possible to assess how much coverage there is. Where
- there are test suites that are written as series of ad hoc
- exercises for the implementation, there's no clear
- specification against which the tests can be examined for
- either completeness or correctness.
-
- >So, if a test suite is only an implementation of the test methods standard
- >and the test methods standard may not completely or correctly test the
- >technology, what good is it?
-
- It's a much better basis for verifying conformance than
- we'd have otherwise. I can pretty confidently guarantee
- that without test methods standards we'd have less
- confidence in our test suites and that implementations
- would be tested less well than they are now.
-
- Rather than compare the POSIX.1 testing situation with that
- for ada, I find it instructive to consider the case of the
- SVVS. This test suite was written to test a somewhat
- underspecified base document without formalizing the test
- assertions. The result was that the test suite itself
- became the de facto standard, with no effective means for
- review of its correctness.
-
- >Even better -- or, more confusing --
- >how do we verify that the test suite is a complete and correct
- >implementation of the test methods standard (which may or may not be
- >complete or correct)??
-
- By reading them carefully, looking at the results when test
- suites written to them are run, and correcting the test
- suites and the test methods and the base specifications
- when we find problems. Just like any other software
- product.
-
- Chuck Karish karish@mindcraft.com
- Mindcraft, Inc. (415) 323-9000 x117
-
- Volume-Number: Volume 31, Number 20
-
-